- Add FA3 > FA2 > SDPA attention backend dispatch
- FA2 wrapper uses @torch.compiler.disable + fullgraph=False
- FA3 uses fullgraph=True (compatible with torch.compile)
- Default FP16_KEEP_NAME_PATTERNS empty (quantize everything, matches PR openai#315)
- Add pod_setup.sh with FA3/FA2 install flow
- Add build_fa3_wheel.sh for pre-building FA3 on cheap 1xH100

Rename folder to today's date. Replace train_gpt.py with the new
baseline from PR openai#315 (11L XSA4 + EMA + Partial RoPE + Late QAT,
1.1248 BPB). Previous script preserved as previous_train_gpt.py.
Update README with PR lineage and new baseline context.

…unner

Port per-head gated attention (12ch, 2*sigmoid) into the PR openai#315
train_gpt.py (11L XSA4 + EMA + Partial RoPE + Late QAT, 1.1248 BPB).
Update run script to use PR openai#315 config for both baseline and experiment.

… log

- Rebased train_gpt.py on PR openai#315 (1.1248 BPB SOTA)
- Added SGD TTT and causal TTT variant
- Added gradient-guided adaptive quantization (int5/int6/int7)
- Added z-loss regularization
- Updated plan with current landscape and run commands

Merged records from all experiment branches into one working branch.
Updated CLAUDE.md with current competitive landscape and next priorities.
Rewrote idea bank with tiered roadmap for closing the gap to openai#315.

torch.compile constant-folds CastedLinear._qat at first trace.
Credit: @152334H via PR openai#315.

torch.compile constant-folds CastedLinear._qat at first trace.
Credit: @152334H via PR openai#315.

torch.compile constant-folds CastedLinear._qat at first trace.
Credit: @152334H via PR openai#315.

13 techniques tested that did NOT work on PR openai#315 base:
- Causal TTT (3 variants): neutral on EMA+XSA base
- MTP: +0.028 BPB, throughput penalty kills it
- INT4: 0.06 BPB quant gap wipes out param advantage
- Canon layers: 48% step overhead not compensated
- Memory tokens, gradient-guided quant, cautious WD,
  L1 regularization, label smoothing, 1M batch, full QAT

4 positive findings:
- EMA > SWA by 0.003 BPB (3-seed verified)
- Weight decay directly controls artifact size
- 786K > 524K batch by 0.004 BPB
- FA3 Hopper: 15-20% more steps at same wallclock

Best verified result: 1.1257 BPB (PR openai#315 reproduction)
Includes 12 training logs for verification.

13 techniques tested that did NOT work on PR openai#315 base:
- Causal TTT (3 variants): neutral on EMA+XSA base
- MTP: +0.028 BPB, throughput penalty kills it
- INT4: 0.06 BPB quant gap wipes out param advantage
- Canon layers: 48% step overhead not compensated
- Memory tokens, gradient-guided quant, cautious WD,
  L1 regularization, label smoothing, 1M batch, full QAT

4 positive findings:
- EMA > SWA by 0.003 BPB (3-seed verified)
- Weight decay directly controls artifact size
- 786K > 524K batch by 0.004 BPB
- FA3 Hopper: 15-20% more steps at same wallclock

Best verified result: 1.1257 BPB (PR openai#315 reproduction)
Includes 12 training logs for verification.

13 techniques tested that did NOT work on PR openai#315 base:
- Causal TTT (3 variants): neutral on EMA+XSA base
- MTP: +0.028 BPB, throughput penalty kills it
- INT4: 0.06 BPB quant gap wipes out param advantage
- Canon layers: 48% step overhead not compensated
- Memory tokens, gradient-guided quant, cautious WD,
  L1 regularization, label smoothing, 1M batch, full QAT

4 positive findings:
- EMA > SWA by 0.003 BPB (3-seed verified)
- Weight decay directly controls artifact size
- 786K > 524K batch by 0.004 BPB
- FA3 Hopper: 15-20% more steps at same wallclock

Best verified result: 1.1257 BPB (PR openai#315 reproduction)
Includes 12 training logs for verification.

Safe config matching PR openai#315 proven techniques:
- 11 layers, MLP 3x (1536), BigramHash 2048
- Muon backend_steps=5, momentum=0.99 (proven by all top PRs)
- XSA on last 4 layers, Partial RoPE 16/64, LN Scale, Late QAT
- EMA decay=0.997 every 4 steps via torch._foreach_lerp_
- CUDA_DEVICE_MAX_CONNECTIONS=1 for multi-GPU overlap
- SmearGate, OrthoInit, int5 MLP/int6 attention, zstd-22

…le, EMA, Late QAT, TTT

Major rewrite targeting top-5 leaderboard:
- 11 layers (from 10), BigramHash reduced to 10240 to fit 16MB
- XSA (Exclusive Self-Attention) on last 4 layers
- Partial RoPE: 16/64 head dims get position encoding
- LN Scale: 1/sqrt(layer+1) dampening on deeper layers
- EMA (decay=0.997) replaces SWA
- Late QAT: STE int6 enabled only in final 4% of training
- TTT: 25-epoch SGD on val data post-quantization
- FA3 auto-detection with SDPA fallback
- Reverted SwiGLU back to relu² (confirmed worse by openai#340, openai#344)

… 3 seeds)

AdamW TTT with cosine lr decay over 30 epochs and per-layer lr groups
(3x for MLP output projections, 0.5x for input projections). 34 TTT
configurations tested. FINDINGS.md documents 31 experiments including
negative results on codebook quantization, symmetry-transport, layer
dropping, focal loss, and KL divergence TTT.

Builds on PRs openai#162, openai#180, openai#77, openai#398, openai#442, openai#417, openai#315.

ROPE_DIMS=16: apply rotary to 25% of head dims, rest position-free
LN_SCALE=1: scale RMSNorm output by 1/sqrt(layer+1)
Both env-var gated, default off — existing runs unaffected.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

ROPE_DIMS=16: apply rotary to 25% of head dims, rest position-free
LN_SCALE=1: scale RMSNorm output by 1/sqrt(layer+1)
Both env-var gated, default off — existing runs unaffected.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…e-lateqat-1.1248

Record: 11L Partial RoPE + LN Scale + EMA + XSA4 (val_bpb: 1.1248)